Hello!
Me (pise@kth.se) and De Chi Hao (dchao@kth.se) have gotten approval from the authors of this PR (over e-mail) that we will give feedback on their finished result. Once this PR has been merged, we will open our own "Feedback Task PR" accordingly.

@PierreSegerstrom Yes, sure. But I think we need to have this PR merged first :D

@frankreyesgarcia any review, please?

@PierreSegerstrom We've finished the tutorial! I'll send you the link by email, and you can start on the feedback.

@JingzeGuo Thank you, we received your email, and I was able to open the Google Colab notebook 🙌🏻

We'll get to it, and get back here with the feedback! (edit: by Wednesday EOD)

Feedback

No Account

The executable tutorial is built on a Google Colab environment, indicating that it is reproducible, as we only need a free Google account to make a copy of the ipynb file. This made it very easy for us to get into, requiring no setup.

One tiny tweak to make it even clearer, is to put the “execution note” at the very beginning, under your names for example. It fits better there, since during reading the “Motivation”, the thought came to mind: “will we need to consider any special setup?”

Executability

The tutorial is fully automated, as running the "Run All" command executes all steps in sequence. The total execution time taken was approximately 7 minutes.

Technical Depth

The tutorial indeed implements a non-trivial DevOps workflow involving integrations such as Node.js CLI(promptfoo), a local Python model interface (Qwen2.5-0.5B-Instruct), declarative test suites and a structural assertions YAML config file, and the GitHub Actions CI automation quality gate.

Relevance

Yes, the authors clearly laid out in the first “Motivation” section. The tutorial mentioned how it reflects the standard DevOps practice of Automated regression testing as a CI quality gate. They are using this concept to address the prompt drift issue, preventing the potential consequences of a broken downstream schema contract and a degradation of output quality.

The authors make a good case for why this is important in the beginning of the tutorial. One possible addition to the “Motivation” section of the notebook to really “hit the nail”, is to also give a quick example for what could go wrong if we allowed “prompt drift” to happen. Answering the question: “Is it really that big of a deal?”. You do show this through the failed tests later on, but a quick justifying sentence in the beginning can be a nice addition.

System Reasoning

The tutorial clearly explained how different components collaborate, including the promptfooconfig.yaml, provider.py, and CLI evaluation tool. It was a nice detail how provider.py acts as an interface, so that it can be easily switched out to a hosted API later.

Design Decisions

The tutorial leaves a markdown chapter, Design Decisions and Tooling Rationale, which clearly shows the reasons for their choice of programming tools and architecture. For example, the reasons for why it chose local open-weights models due to its no API costs and reproducibility, etc. The choice of using examples from a standard dataset was a nice touch!

Reflection

The tutorial markdown chapter titled Who This Fits, Limitations and Practical Considerations clearly reflects on the cases in which this tutorial approach can be useful, which other two situations might not, and its Limitations and Practical Considerations. This section about “applicable scenarios” was particularly useful to understand the application this. While reading, we already envisioned how this could be applied in a hypothetical LLMOps pipeline.

Narrative/Structure

The overall structure was concise and clear. It follows a standard timeline from Motivation, Setup, Steps, Outcome, and reflection. Even though there are multiple steps to follow, we find it structured in a way that we don’t lose track of the bigger picture. we also found it good that you first showed the “good example” and explained it well, so when the “bad example” was introduced, we were encouraged to reflect upon what would now change.

Visuals

The tutorial uses the Mermaid architecture diagram and includes a bar chart summarizing test pass and fail statistics using matplotlib. Although the output from “promptfoo” itself was enough to understand what happened in the tests, the added graph visualization in step 5 conceptually guided me as a reader that we would go from “good example” (green) to “bad example” (red).

Language

The overall language used in this article was well written, clear, and structured. A great addition is that some of the keywords were highlighted in bold text, which made it easier for me to understand.

ILO

The intended learning outcomes are clearly stated at the top of the tutorial. After completing the tutorial, four LLMOps skills can be introduced. While reading the ILOs, We felt encouraged to carry out the tutorial, since I’m presented with the actual outcomes of the tutorial (what kinds of insights/knowledge/learnings we can carry with us afterwards). It is common that ILOs are phrased very “high level”, which can make the reader less engaged since they aren’t concrete enough. You’ve planned out the contents of this tutorial well, nice job!

Certification

I/We certify that generative AI, incl. ChatGPT, has not been used to write this feedback. Using generative AI without permission is considered academic misconduct.

De Chi Hao (dchao@kth.se) and Pierre Segerström (pise@kth.se)

@PierreSegerstrom Thanks for the detailed feedback! We’ll revise it following your suggestions in the final version.